The difficulty of making Multiple Virtual Storage (MVS) files available to other systems is well known in the art. There are two aspects to making count, key and data (CKD) format files from a MVS available to a system which employs Fixed Block Architecture files: getting the data from one system to another in a form allowing sharing, and recoding it from one system's conventions to the other.
The files may be transferred, rather than shared, by using a simple text transfer, for example, by using an internet binary image mode, which is stream binary. Each internet connection sets up a separate connection and this is extremely inefficient and error prone. The form of transfer does not accomplish sharing of the data sets.
A method for transferring large data sets to another system is to write a tape and move the tape to the other system for reading, using one tape drive that is compatible with the MVS system and another tape drive that can read the same tape, but which is compatible with the other system. The data sets on each system after the transfer are totally separate and are not shared.
Utilities may be able to edit the data to convert text files, but are handled very inefficiently and transferred as a single file, and the transfer may not be error free.
Data mining is the ability to analyze a data file to interpret the file and ascertain information that is not apparent from a direct reading of the file. Examples include analyzing credit card data to ascertain the most likely group of persons who spend money more than a particular amount by credit card a month based on deposit and age, and to then select a more detailed group who have deposits, for example, between $3000 and $4000 and are in their 40's.
Another example is to ascertain unusual credit card activity and thereby identify possible credit card fraud.
In order to perform an accurate analysis, error free data is required.
In order to provide an effective analysis to, for example, identify a compromised credit card promptly, the access to the data set must be prompt.
Lastly, the data set to be analyzed may be changing rapidly so that it would be advantageous to provide continuing updates to the file as it is being analyzed.
Most credit card systems that would be analyzed are MVS systems. Most data mining applications are done by data processors running UNIX or AIX which employ FBA data, which data is incompatible with the CKD data of the MVS systems. The typical data set to be analyzed is in the range of 10's or 100's of gigabytes, which would require many hours of transmission time between systems using communication techniques.
Several techniques have been developed for emulating CKD on FBA but typically are usable for only small data sets or a limited number of records, and/or do not allow access by an FBA device, and therefore do not allow sharing of the data. Examples are U.S. Pat. No. 5,206,939 to Yanai et al. which discloses a compression technique for representing the COUNT field of every record in a cascading arrangement, U.S. Pat. No. 5,301,304 to Menon which discloses an emulation program which completely emulates CKD data stored on FBA devices while still retaining the same byte displacements of the CKD data by elimination of padding, intra-record gaps, etc., U.S. Pat. No. 5,283,884 to Menon et al. which stores a table entry in non-volatile memory for each record, and U.S. Pat. No. 5,535,372 to Benhase et al. which provides a track format descriptor in non-volatile storage for each track and describes whether the track comprises "well behaved" formatted data which is simple to search or is not "well behaved" such that the entire track must be loaded into cache.
It may be desirable to transfer the results of the data mining back to the originating data processing system to identify the specific accounts meeting the criteria of the analysis so that specific mailings may be conducted or phone calls placed, etc. Thus, a user requirement may be that access to the data set must be maintained by the originating system for proper coordination.